<html>
  <head>
    <meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Logstash快速入门 | 少数派报告</title>
<meta name="description" content="Minority Report" />
<link rel="shortcut icon" href="https://www.timegarage.works/favicon.ico?v=1598261443340">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.10.0/katex.min.css">
<link rel="stylesheet" href="https://www.timegarage.works/styles/main.css">

<script src="https://cdn.bootcss.com/highlight.js/9.12.0/highlight.min.js"></script>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Droid+Serif:400,700">



  </head>
  <body>
    <div class="main">
      <div class="main-content">
        <div class="site-header">
  <a href="https://www.timegarage.works">
  <img class="avatar" src="https://www.timegarage.works/images/avatar.png?v=1598261443340" alt="">
  </a>
  <h1 class="site-title">
    少数派报告
  </h1>
  <p class="site-description">
    Minority Report
  </p>
  <div class="menu-container">
    
      
        <a href="/" class="menu">
          首页
        </a>
      
    
      
        <a href="/archives" class="menu">
          归档
        </a>
      
    
      
        <a href="/tags" class="menu">
          标签
        </a>
      
    
      
        <a href="/post/Portfolio/" class="menu">
          作品
        </a>
      
    
      
        <a href="/post/about/" class="menu">
          关于
        </a>
      
    
  </div>
  <div class="social-container">
    
      
    
      
    
      
    
      
    
      
    
  </div>
</div>

        <div class="post-detail">
          <article class="post">
            <h2 class="post-title">
              Logstash快速入门
            </h2>
            <div class="post-info">
              <span>
                2020-07-13
              </span>
              <span>
                3 min read
              </span>
              
                <a href="https://www.timegarage.works/tag/ELK/" class="post-tag">
                  # ELK
                </a>
              
            </div>
            
              <img class="post-feature-image" src="https://www.timegarage.works/post-images/LogParsing.png" alt="">
            
            <div class="post-content-wrapper">
              <div class="post-content">
                <center>最近在网络流量分析项目中需要对日志进行收集和处理，而网络上关于Logstash的资料都比较分散，因此本文对Logstash的架构与用法进行了简单地梳理，帮助初学者快速入门。</center>
<!-- more -->
<h2 id="简介">简介</h2>
<p>Logstash是开源的服务器端数据处理管道，能够同时从多个来源采集数据，处理数据和转发数据。</p>
<h2 id="工作流程">工作流程</h2>
<p>数据输入 &gt; 数据过滤 &gt; 数据输出</p>
<h2 id="关键组件">关键组件</h2>
<p>Logstash使用input，filter，output三个组件分别完成数据采集、处理和转发的工作。</p>
<p><b>input组件负责采集数据各种格式、大小和来源的数据。</b>input组件能够以连续流式传输的方式，轻松地从日志、指标、Web应用、数据存储以及各种AWS服务采集数据。</p>
<p><b>filter插件负责动态地转换和解析数据，不受格式或复杂度的影响。</b>Grok插件可以帮助开发者方便地从非结构化数据中解析出结构化数据。</p>
<p><b>output插件负责转发filter处理过的数据到指定的数据库中。</b>Logstash提供了众多的输出选择，开发者可以将数据发送到指定的地方。例如，可以用elasticsearch插件输出到es ，rediss插件输出到redis，stdout插件标准输出，kafka插件输出到kafka等。</p>
<p>此外，Logstash具有高度的可扩展性，开发者可以根据自身需要定制插件。</p>
<h2 id="grok">Grok</h2>
<p>Grok是Logstash最重要的插件之一，使用正则表达式对数据完成过滤和解析工作。</p>
<p>参考资料：<a href="https://blog.csdn.net/qq_34021712/article/details/79746413">Logstash Grok插件语法介绍 </a></p>
<h2 id="filebeat">Filebeat</h2>
<p>Filebeat是用来替代Logstash Forwarder的下一代Logstash收集器，与Logstash相比更加轻量和高效，它可以很方便地与Logstash或Elasticsearch进行对接。</p>
<p><b>Filebeat 与 Logstash的联系与区别：</b></p>
<p><b>联系：</b>Filebeat和Logstash都具有日志收集的功能</p>
<p><b>区别：</b>Filebeat更轻量，占用资源较Logstash更少。Logstash具备过滤和处理数据的功能，但Filebeat只能采集日志。</p>
<p>在实际使用中，通常使用Filebeat作为数据收集的工具，接着使用Redis/ Kafka / RabbitMQ等缓存数据，最后将数据输出到Logstash完成数据聚合、处理的工作。</p>

              </div>
              <div class="toc-container">
                <ul class="markdownIt-TOC">
<li>
<ul>
<li><a href="#%E7%AE%80%E4%BB%8B">简介</a></li>
<li><a href="#%E5%B7%A5%E4%BD%9C%E6%B5%81%E7%A8%8B">工作流程</a></li>
<li><a href="#%E5%85%B3%E9%94%AE%E7%BB%84%E4%BB%B6">关键组件</a></li>
<li><a href="#grok">Grok</a></li>
<li><a href="#filebeat">Filebeat</a></li>
</ul>
</li>
</ul>

              </div>
            </div>
          </article>
        </div>

        

        
          
            <link rel="stylesheet" href="https://unpkg.com/gitalk/dist/gitalk.css">
<script src="https://unpkg.com/gitalk/dist/gitalk.min.js"></script>

<div id="gitalk-container"></div>

<script>

  var gitalk = new Gitalk({
    clientID: '447058976c8bd4fe04f6',
    clientSecret: 'b4a632203857b02425883cbf9b5bea3d5a2f86b0',
    repo: 'timegarage.github.io',
    owner: 'TimeGarage',
    admin: ['TimeGarage'],
    id: (location.pathname).substring(0, 49),      // Ensure uniqueness and length less than 50
    distractionFreeMode: false  // Facebook-like distraction free mode
  })

  gitalk.render('gitalk-container')

</script>

          

          
        

        <div class="site-footer">
  Copyright © 2020 TimeGarage Inc. | 
  <a class="rss" href="https://www.timegarage.works/atom.xml" target="_blank">RSS</a>
</div>

<script>
  hljs.initHighlightingOnLoad()

  let mainNavLinks = document.querySelectorAll(".markdownIt-TOC a");

  // This should probably be throttled.
  // Especially because it triggers during smooth scrolling.
  // https://lodash.com/docs/4.17.10#throttle
  // You could do like...
  // window.addEventListener("scroll", () => {
  //    _.throttle(doThatStuff, 100);
  // });
  // Only not doing it here to keep this Pen dependency-free.

  window.addEventListener("scroll", event => {
    let fromTop = window.scrollY;

    mainNavLinks.forEach((link, index) => {
      let section = document.getElementById(decodeURI(link.hash).substring(1));
      let nextSection = null
      if (mainNavLinks[index + 1]) {
        nextSection = document.getElementById(decodeURI(mainNavLinks[index + 1].hash).substring(1));
      }
      if (section.offsetTop <= fromTop) {
        if (nextSection) {
          if (nextSection.offsetTop > fromTop) {
            link.classList.add("current");
          } else {
            link.classList.remove("current");    
          }
        } else {
          link.classList.add("current");
        }
      } else {
        link.classList.remove("current");
      }
    });
  });

</script>

      </div>
    </div>
  </body>
</html>
